AFSD: Adaptive Feature Space Distillation for Distributed Deep Learning

We propose a novel and adaptive feature space distillation method (AFSD) to reduce the communication overhead among distributed computers. The proposed improves Codistillation process by supporting longer update interval rates. AFSD performs knowledge distillates across models infrequently provides flexibility in terms of exploring diverse variations training process. perform sharing instead output only. Therefore, we also new loss function for technique AFSD. Using leads more efficient transfer between with In our method, can achieve same accuracy as Allreduce fewer epochs.